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SLIDE 1: DEVELOPING A SPOT CRM 
DEBRIEFING TOOL - 
BACKGROUND TO THE WORK 
In a study of CRM LOFT debriefings 
published in 1997, Dismukes, McDonnell 
& Jobe reported that debriefings were not 
being utilized as fully as they could be 
and that crews may not be getting the full 
benefit from LOFT that is possible. On 
the basis of their findings, they suggested 
a set of general guidelines for debriefings 
for the industry. 

Our work builds on this study to try to 
provide a specific debriefing tool which 
provides a focus for the strategies that 
Dismukes et al suggest. 


SLIDE 2: THE LEARNING CYCLE 
The importance of debriefing within a 
training programme is illustrated by the 
learning cycle proposed by Mayes et al in 
1994. Whilst presentation of the theory 
informs the learner, and practicing gives 
the learner experience, it is the debrief, or 
subsequent feedback session which helps 
the learner to make sense of their actions 
and to tie ‘what they did’ to what they 
were taught. Despite being last, the 
debrief session is certainly not the least 
element of the learning cycle! However, 
it is arguably the hardest to complete 
successfully because the instructor has to 
make the connection between the theory 
and practice, and not just general practice 
but that of the participating crew. 

The aim of our debriefing tool was to 
assist the instructor to make the 
connections between concrete actions and 
the principles that the airline wants their 
crews to work by; to provide a means to 
bridge theory and action. 





Training Focus 

• Airline has 6 CRM categories: 

- Planning - Situation Awareness 

- Decision Making - Communication 

- Workload - Crew Coordination 

Management 

• Focus of scenario studied is situation 
awareness management 



Simulation Scenario 


,V Fly Direct ro Broadway 
Pax gea up to to to lavatory 
“Climb and maintain 14.000 feet" 


"Stop your climb ■ 12.000 feet' 
Altitude capture Jails 



Broadway 

(BWZ) 



I Preflighi 

FIO Nov Ditplay Inoperative 
Anti-ice requited 
Preflight Checklists 


2 Runway I V Whncslooe Climb 
Turn right heading 173 
At 2.3 DME. mm left to 040 
Maintain 3.000 feet 
Do not exceed 210 Iris 




SLIDE 3 - 5: THE SCENARIO 
For the participating airline the principles 
we were concerned with were their six 
CRM categories - planning, SA, DM, 
communication, workload management 
and coordination. They select one of 
these categories as a focus each year. In 
1997 this was situation awareness and its 
management. 

The category is presented in a 
groundschool session at the beginning of 
die crews’ recurrent training - which is a 
yearly event. 

The rest of the CRM training occurs on 
the second day of the recurrent in a SPOT 
session. SPOT is Special Purpose 
Operational Training where the simulation 
crews participate in may or may not be a 
complete trip. In 1997 the trip was a 
flight from LGA to DFW but the cruise 
portion was omitted in the interests of 
time and the simulator repositioned as the 
flight went into cruise. The SPOT 
practical is combined with a maneuvers 
practical and the two sessions are 
debriefed after. 

The scenario involves preparation and 
departure from LGA - a complicated 
departure with many restrictions that the 
crew has to observe, followed by an 
automation failure where the VNAV 
capture does not work. The second 
portion of the scenario involves a 
‘typical’ approach to DFW with two 
runway changes. It is aborted at the last 
minute due to traffic on the runway. As 
the crew goes around the flight attendants 
report smoke in the cabin and the crew 
has to bring the airplane in on an 
emergency landing. 

Our aim was to identify important 
behaviours from an SA perspective 
within this scenario which the instructors 
could use as prompts in the debrief for 
both themselves and to encourage crews 
to discuss their SA. 
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Simulation Scenario (cont'd) 


< < . , if>m Smokr/Puy 

Right attendant reports smoke in cabin 

Land ASAP 

Don oxygen masks and goggles 
Maintain contact with FA over interphone 
Conduct abbreviated checklists 
Finish Before Landing checklist 
initiate Emergency Evacuation 


U&2LX jfeft A caU M l 
At 200 feet told to go around 
due to timffic crossing runway 


6. seen od Kunw, 

“Geared for the ILS 36L approach'* 

At 800 feet “You’re too close to the 
traffic ahead. Riwway 36R is available" 


4 Bonham Arrival 
Expect vectors to runway 15C 


Bonham 

(BYP) 



Dallas - Ft Worth 


?,.PW Rupwqy CftWB 

Chiajc year runway to ML" 

Ciew mM react pond and brief the new ap proac h 


T riangulationMethod 

• Video Tapes 

- 11 samples collected 

- Identified behaviors 

- Sorted behaviors for unusual & important 

• Interviews - 5 highly structured, in-depth 
interviews were conducted 

• Theory 



SLIDE 6: METHOD 
We identified behaviours using a 
triangulation method of video, interviews 
and the literature. 

We took the studies of SA from the 
literature, for example Endsley’s and 
Brannick & Princes’, and reviewed these 
in the light of the generic behaviours the 
airline notes as part of their CRM skills. 
For example one SA behaviour is: 
“recognize when automation is becoming 
a detriment to SA and use a more 
appropriate level of automation”. We 
identified instances within the SPOT 
scenario where we thought these points 
may apply and therefore the places in the 
scenario where SA was particularly 
critical. 

Our second method was to interview five 
instructors in depth about the behaviours 
they were looking for in the SPOT and 
why. As part of the interview, we asked 
instructors to rank the SPOT events in 
terms of their importance. They rated the 
departure briefing, altitude capture fail, 
approaches and dealing with die smoke as 
the four most important events, and on 
the basis of this we selected these four 
events for us to focus our efforts on. 

The third method was to analyze the 
behaviours of crews in video tapes of 1 1 
SPOT sessions. These were regular line 
crews who were undergoing their 
recurrent training and agreed for us to 
analyse the video. We listed all the 
behaviours we saw on video, and then 
using our findings from the first two 
methods, we extracted the relevant 
behaviours. 



"Real World" Considerations 

• Keep it short and simple 

• Rapidly reconfigurable event sets 

• Useful to structure discussion 

• Useful as an aide memoire 
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NASA Am i*H 

Example Items From Checklist 

• Aware of traffic (SA) 


SLIDE?: DEVISING THE 
CHECKLIST 
In addition to the theoretical 
considerations of behaviours and 
principles, we had to take into account the 
practical setting in which the training 
takes place and our objectives. 

• During the interviews instructors 
stressed how busy they were during 
the SPOT and the need for any tool to 
be short and simple. The group is 
tired after a four hour session and 
only has a limited amount of 
concentration left - a tool should not 
be too long. 

• From our perspective, we wanted a 
tool that would assist the instructor to 
structure discussion rather than be 
prescriptive to allow for the individual 
differences between crews. 

• Many airlines are moving towards 
compiling SPOTs from a series of 
rapidly reconfigurable event sets, 
meaning the tool would have a longer 
usefulness if the items could be 
clearly divided amongst the events. 


• Monitor altimeter (SA) 

• Note cues to catch altitude capture 
failure (SA) 

• Pilot levels manually prior to reaching 
12,000 feet (DM) 

• Set up autopilot to test (P) 



SLIDE 8: EXAMPLE LIST OF ITEMS 
From our approach, we developed a tool 
which contains 30 items under the four 
event sets. 

This example shows the behaviours we 
selected for the failed altitude capture 
event. In addition to the five items we 
listed the CRM behaviour that these 
actions reflected. The intention was that 
instructors would be able to raise these 
behaviours and discuss with the crew 
how they reflected that the crew had, or 
did not have, SA, or a plan at that point. 
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What They Discussed 

• 15 audio tapes of debriefings 

• Compared to debriefing tool 

• 4 events discussed 



Events Most Frequently Discussed 




SLIDE 9: WHAT THEY TALKED 
ABOUT 

Due to some unfortunate circumstances, 
we were unable to test the use of the tool 
in the debriefing sessions. However, we 
were able to collect 15 audio tapes of 
debriefing sessions. We content analyzed 
these sessions and compared the 
debriefing themes with the tool items to 
assess whether they would have 
provided a useful structure for the 
discussions. 

On average instructors discussed around 
four of the events from the SPOT in their 
debriefings. Among the most frequently 
discussed, were two of the four events 
we had selected: the failure of altitude 
capture and the smoke in the cabin - both 
of which were talked about in around 9 
out of 15 debriefs or 60% of the time. 

The runway changes and preflight brief 
were discussed less often. 

The most popular topics for discussion 
were the distractions on the climb - such 
as traffic; the way the crews used the 
automation, to their advantage or 
disadvantage; the way they divided their 
workload and how hurried rushed or 
busy they felt. 

All of the debriefs included mention of 
SA, although the depth of discussion 
varied markedly - the number of mentions 
varying between 2 and 1 0. Of the 
remaining CRM skills, workload 
management was discussed most often, 
being mentioned in 1 3 of the 1 5 debriefs - 
that’s 87% - and just under 3 times in a 
session on average. 



t 


Most Frequently Covered Tool Items 



SLIDE 10: VERIFYING THE 
BEHAVIOURS 

Against this background view of the 
discussion we compared the items on the 
tool to see whether they would have been 
relevant to mention. 21 out of the 30 
items were covered in at least one of the 
discussions, with the most often 
discussed behaviour being “monitoring 
the altimeter” in the altitude capture failure 
- discussed in 9 out of 15 or 60% of the 
debriefs. Another frequently discussed 
item is “deciding the level of automation 
to use” during the smoke incident - 
discussed in 8 out of 15 or 53% of the 
debriefs. 

A distinction we had to make when 
looking at whether the tool could have 
been used in the debreifs was between the 
instructors mentioning an event versus 
discussin the event in depth. Whilst 
altitude capture was mentioned the most 
often in debreifs - by 14 /15 or 93% of 
instructors, on average they only talked 
about one or two of the items (ave 1 .66) 
when they did. The smoke event, as a 
contrast, was discussed by fewer 
instructors - by 1 1/15 or 79%, but they 
covered the event in more depth, 
discussing aropund three of the items - on 
average 2.66. 

What we studied here is the unknowing 
coverage of the tool items with respect to 
the themes that were discussed, to assess 
whether the tool would have been any use 
to the instructor. Our conclusions 
therefore are somewhat speculative but 
what they did do was provide us with a 
methodology and a way forward for our 
continuing study this year. 



Conclusions 
30 items is too many 

Important ;* discussed 

Link SA to actions 



SLIDE 1 1: CONCLUSIONS 
ft seems that even 30 items spread over 
four events is still too many - instructors 
can only make a limited number of points 
in the time and crew attention span that 
they have available. The tool needed to 
have fewer items - an upper limit of 
around 20 rather than 30. 

It seems that our focus on the DFW 
runway change & preflight events were 
not appropriate, despite instructors rating 
these as important SA events. We learnt 
that some of the events that are important 
are generally handled well by the crews 
and therefore cause little discussion when 
there are other less certain events to 
debate. 

Despite citing certain events as important, 
the instructors do not always discuss 
them. In particular, the preflight brief 
gets far less discussion time than the 
instructors’ importance rating would 
suggest it should have. This may be 
because the crews are not really ‘doing’ 
anything but are thinking, and instructors 
find it hard to make the connection 
between SA as reflected in crew 
discussion versus physical actions where 
the connection can be more easily 
described. 

We concluded that in this respect the tool 
would not have assisted the instructor. 
Whilst it identified the behaviours that 
could be connected with SA and other 
CRM skills, and thus sifted out a myriad 
of other behaviors which could be 
discussed and provided more of a focus 
on SA, it did not help the instructor make 
the connection by providing the link. 

For the next tool, which is now in 
development, we need to provide 
instructors with that explicit SA 
connection between the selected 
behaviour and the CRM skill. For 
example, the behaviour “noting the cues 
that the altitude is not going to capture” 
involves the monitoring component of 
S A. If a crew notices these cues they will 
be able to act before the capture fails and 
thus save themselves the added workload 
of trying to regain their altitude. 



